Methods and apparatus for providing a frame packing arrangement for panoramic content

ABSTRACT

Apparatus and methods for providing a frame packing arrangement for the encoding/decoding of, for example, panoramic content. In one embodiment, an encoder apparatus is disclosed. In a variant, the encoder apparatus is configured to encode Segmented Sphere Projections (SSP) imaging data and/or Rotated Sphere Projections (RSP) imaging data into an extant imaging format. In another variant, a decoder apparatus is disclosed. In one embodiment, the decoder apparatus is configured to decode SSP imaging data and/or RSP imaging data from an extant imaging format. Computing devices, computer-readable storage apparatus, integrated circuits and methods for using the aforementioned encoder and decoder are also disclosed.

PRIORITY

This application claims the benefit of priority to U.S. ProvisionalPatent Application Ser. No. 62/477,936 filed Mar. 28, 2017 of the sametitle; U.S. Provisional Patent Application Ser. No. 62/473,952 filedMar. 20, 2017 of the same title; U.S. Provisional Patent ApplicationSer. No. 62/465,678 filed Mar. 1, 2017 of the same title; U.S.Provisional Patent Application Ser. No. 62/462,804 filed Feb. 23, 2017of the same title; and U.S. Provisional Patent Application Ser. No.62/446,297 filed Jan. 13, 2017 and entitled “Methods and Apparatus forRotated Sphere Projections”, each of the foregoing being incorporatedherein by reference in its entirety.

RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.15/289,851 filed Oct. 10, 2016 and entitled “Apparatus and Methods forthe Optimal Stitch Zone Calculation of a Generated Projection of aSpherical Image”, which is incorporated herein by reference in itsentirety.

This application is also related to U.S. patent application Ser. No.15/234,869 filed Aug. 11, 2016 and entitled “Equatorial Stitching ofHemispherical Images in a Spherical Image Capture System”, which claimsthe benefit of priority to U.S. Provisional Patent Application Ser. No.62/204,290 filed on Aug. 12, 2015, each of the foregoing beingincorporated herein by reference in its entirety.

This application is also related to U.S. patent application Ser. No.15/406,175 filed Jan. 13, 2017 and entitled “Apparatus and Methods forthe Storage of Overlapping Regions of Imaging Data for the Generation ofOptimized Stitched Images”, which is also incorporated herein byreference in its entirety.

COPYRIGHT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The present disclosure relates generally to video image processing andin one exemplary aspect, to methods and apparatus for providing a framepacking arrangement for panoramic, 360° or virtual reality (VR) imagesusing, for example, extant codecs.

Description of Related Art

Panoramic images (e.g., spherical images) are typically obtained bycapturing multiple images with overlapping fields of view from differentcameras and combining (“stitching”) these images together in order toprovide, for example, a two-dimensional projection for use with moderndisplay devices. Converting a panoramic image to a two-dimensionalprojection format can introduce some amount of distortion and/or affectthe subsequent imaging data. However, two-dimensional projections aredesirable for compatibility with existing image processing techniquesand also for most user applications. In particular, many encoders andcompression techniques assume traditional rectangular image formats.

Incipient interest into different projections and applications hassparked research into a number of possible projection formats. Examplesof prior art projection formats include without limitation e.g.,equirectangular, cubemap, equal-area, octahedron, icosahedron, truncatedsquare pyramid, and segmented sphere projection. For each of theseprojection formats, multiple facet (also called frame packing)arrangements are possible. A selection of prior art projections aredescribed within e.g., “AHG8: Algorithm description of projection formatconversion in 360Lib”, published Jan. 6, 2017, to the Joint VideoExploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG11, the contents of which being incorporated herein by reference in itsentirety.

While techniques exist that enable the encoding/decoding of thisso-called panoramic content, extant frame packing arrangement techniquesfor these panoramic images may prove sub-optimal, especially in thecontext of pre-existing codecs. For example, the encoding/decoding ofpanoramic images using pre-existing codecs may result in, inter alia,increased processing overhead, lack of adequate bandwidth (bitrate)considerations, decreased compression efficiencies, lack of resolutionor high battery utilization associated with the encoding and decodingprocesses.

SUMMARY

The present disclosure satisfies the foregoing needs by providing, interalia, methods and apparatus for providing a frame packing arrangementfor panoramic images using, inter alia, extant codecs.

In one aspect, an encoder apparatus is disclosed. In one embodiment, theencoder apparatus is configured to encode Segmented Sphere Projections(SSP) imaging data and/or Rotated Sphere Projections (RSP) imaging datainto an extant imaging format.

In one variant, the encoder apparatus is configured to encode SSPimaging data and/or RSP imaging data into a pixel width of the extantimaging format.

In another variant, the encoder apparatus is configured to encode anequirectangular projection (ERP) into a first pixel width of the extantimaging format and encode either fisheye projection imaging data or RSPpolar imaging data into a second pixel width of the extant imagingformat.

In yet another variant, the fisheye projection imaging data or RSP polarimaging data is segmented and disposed within the second pixel width ofthe extant imaging format.

In yet another variant, the fisheye projection imaging data or RSP polarimaging data is segmented into two halves of circle, with the two halvesof the circle disposed adjacent to one another in the second pixelwidth.

In yet another variant, the hemispherical half of the segmented halvesis disposed along a bottom portion of the second pixel width furthestfrom the first pixel width.

In yet another variant, the hemispherical half of the segmented halvesis disposed along a top portion of the second pixel width closest to thefirst pixel width.

In yet another variant, the hemispherical half of the segmented halvesis disposed such that the hemispherical halves alternate in orientationwithin the second pixel width.

In yet another variant, the imaging data includes stereo imaging data,the stereo imaging data including left imaging data and right imagingdata.

In yet another variant, the stereo imaging data further includes leftand right polar imaging data.

In yet another variant, the extant imaging format includes a pixel widthand a pixel depth, the stereo imaging format configured to run an extentof the pixel width.

In yet another variant, a frame packing arrangement for the extantimaging format first includes left imaging data, followed by left polarimaging data, followed by right imaging data, followed by right polarimaging data.

In yet another variant, the right imaging data is inverted and reversedwith respect to the left imaging data.

In yet another variant, the right imaging data is only inverted withrespect to the left imaging data.

In yet another variant, the right imaging data is only reversed withrespect to the left imaging data.

In yet another variant, the left and right polar imaging data issubstantially identical, and the frame packing arrangement is configuredto obviate either of the left or the right polar imaging data inresponse to a received signal.

In yet another variant, the extant imaging format includes a 4K imagingformat having a pixel width of 4,096 pixels and a pixel depth up to2,048 pixels.

In yet another variant, the encoded image includes an RSP-3×2 or RSP-2×3imaging format.

In another embodiment, the encoder apparatus includes an image capturedevice, the image capture device configured to capture panoramiccontent; a stitching module configured to generate a first projectionthat includes a front panoramic portion, a right-side panoramic portion,and a left-side panoramic portion from the captured panoramic content,the stitching module further configured to generate a second projectionthat includes a back panoramic portion, a top panoramic portion, and abottom panoramic portion from the captured panoramic content; a firstencoder configured to encode the first projection; and a second encoderconfigured to encode the second projection.

In one variant, the first projection is representative of a firstcontinuous portion of the captured panoramic content.

In another variant, the second projection is representative of a secondcontinuous portion of the captured panoramic content.

In yet another variant, the generated first projection includesredundant imaging data with the generated second projection and theencoder apparatus is further configured to black out a portion of theredundant imaging data, the blacked out portion configured to reduce anumber of pixels associated with a combined first projection and thesecond projection as compared with a non-blacked out combined firstprojection and the second projection.

In yet another variant, the encoder apparatus is further configured toinsert metadata information in the blacked out portion.

In yet another variant, the inserted metadata information is utilizedfor the stitching of the captured panoramic content for display on acomputing device.

In yet another variant, the inserted metadata information is utilizedfor determination of a particular encoding format configuration of aplurality of encoding format configurations.

In a second aspect, a decoder apparatus is disclosed. In one embodiment,the decoder apparatus is configured to decode SSP imaging data and/orRSP imaging data from an extant imaging format.

In one variant, the decoded image includes an RSP-3×2 or RSP-2×3 imagingformat.

In a third aspect, and encoding/decoding apparatus is disclosed. In oneembodiment, the encoding/decoding apparatus includes an image capturedevice, a stitching module, two or more encoders, two or more decoders,a transmission line, a reconstruction module and a display device.

In a fourth aspect, a method for encoding imaging data is disclosed. Inone embodiment, the method includes encoding SSP imaging data and/or RSPimaging data into an extant imaging format.

In another embodiment, the imaging data includes a panoramic image andthe method further includes obtaining a first equirectangular projectionthat includes a front panoramic portion, a right-side panoramic portion,and a left-side panoramic portion; cropping the first equirectangularprojection to create a first cropped portion; obtaining a secondprojection that includes a back panoramic portion, a top panoramicportion, and a bottom panoramic portion; cropping the second projectionto create a second cropped portion; and combining the first croppedportion with the second cropped portion in order to create a panoramicprojection.

In one variant, the method further includes receiving a viewportposition for the panoramic projection, the viewport position beingindicative of a portion of the panoramic projection; determining thatthe entirety of the viewport position is located in either the firstcropped portion or the second cropped portion; decoding either the firstcropped portion or the second cropped portion based on the determining;and transmitting either the decoded first cropped portion or the decodedsecond cropped portion.

In another variant, the method further includes causing the display ofeither the transmitted decoded first cropped portion or the transmitteddecoded second cropped portion.

In yet another variant, the method further includes blacking outportions of the panoramic projection, the blacked out portions of thepanoramic projection comprising redundant imaging data.

In yet another variant, the blacking out portions of the panoramicprojection includes blacking out corners of the first cropped portionand the second cropped portion.

In yet another variant, the blacking out portions of the panoramicprojection includes blacking out portions internal to corners of thefirst cropped portion and the second cropped portion.

In yet another variant, the blacking out portions of the panoramicprojection includes only blacking out the second cropped portion of thepanoramic projection, while not blacking out the first cropped portionof the panoramic projection.

In yet another variant, the method further includes inserting metadatainformation into the blacked out portions of the panoramic projection.

In yet another variant, the method further includes an RSP-3×2 orRSP-2×3 imaging format.

In a fifth aspect, a method of apportioning black areas within a targetprojection is disclosed.

In a sixth aspect, a method for creating a target panoramic projectionis disclosed. In one embodiment, the method includes obtaining panoramiccontent in a source projection, splitting the obtained panoramic contentinto a first portion and a second portion, selecting a first targetprojection for the first portion and cropping the first targetprojection, selecting a second target projection for the second portionand cropping the second target projection, and combining the firstcropped portion with the second cropped portion in order to create atarget panoramic projection.

In a seventh aspect, a method for decoding and displaying a targetpanoramic projection is disclosed. In one embodiment, the methodincludes receiving the target panoramic projection that includes a firstcropped portion and a second cropped portion, receiving a viewportposition associated with the target panoramic projection, decoding thefirst cropped portion and/or the second cropped portion in accordancewith the received viewport position, transmitting the decoded firstcropped portion and/or the decoded second cropped portion, and causingthe display of the transmitted portion(s) on a display device.

In an eight aspect, a method for adjusting the quantization parameter ofone or both of the two image facets of a rotated sphere projection isdisclosed. In one embodiment, the method includes obtaining panoramiccontent in a rotated sphere projection, determining whether to adjust aquantization parameter for one of the two image facets for the rotatedsphere projection, determining whether to adjust a quantizationparameter for the other one of the two image facets for the rotatedsphere projection and adjusting one or both of the two image facetsquantization parameter.

In a ninth aspect, a method for transmitting panoramic content in arotated sphere projection that has been optimized for streamingapplications is disclosed. In one embodiment, the method includesobtaining panoramic content in a rotated sphere projection, determiningwhether or not to downsample one or more portions of the rotated sphereprojection, determining whether or not to rearrange the downsampled oneor more portions, and transmitting the content in the rotated sphereprojection.

In a tenth aspect, a method for decoding imaging data is disclosed. Inone embodiment, the method includes decoding SSP imaging data and/or RSPimaging data from an extant imaging format.

In one variant, the method includes an RSP-3×2 or RSP-2×3 imagingformat.

In an eleventh aspect, a computer-readable storage apparatus isdisclosed. In one embodiment, the computer-readable storage apparatusincludes a storage medium comprising computer-readable instructions, thecomputer-readable instructions being configured to, when executed by aprocessor apparatus, to perform at least a portion of the aforementionedmethodologies described herein.

In another embodiment, the computer-readable storage apparatus includesa storage medium comprising computer-readable instructions, thecomputer-readable instructions being configured to, when executed by aprocessor apparatus to decode SSP imaging data and/or RSP imaging datafrom an extant imaging format.

In yet another embodiment, the computer-readable storage apparatusincludes a storage medium comprising computer-readable instructions, thecomputer-readable instructions being configured to, when executed by aprocessor apparatus: obtain a first projection that includes a frontpanoramic portion, a right-side panoramic portion, and a left-sidepanoramic portion; obtain a second projection that includes a backpanoramic portion, a top panoramic portion, and a bottom panoramicportion; and combine the first projection with the second projection inorder to create a panoramic projection.

In one variant, the computer-readable instructions are furtherconfigured to, when executed by the processor apparatus: receive aviewport position for the panoramic projection, the viewport positionbeing indicative of a portion of the panoramic projection; determinethat the entirety of the viewport position is located in either thefirst projection or the second projection; decode either the firstprojection or the second cropped portion based on the determination; andtransmit either the decoded first projection or the decoded secondprojection.

In another variant, the computer-readable instructions are furtherconfigured to, when executed by the processor apparatus: cause thedisplay of either the transmitted decoded first projection or thetransmitted decoded second projection.

In yet another variant, the computer-readable instructions are furtherconfigured to, when executed by the processor apparatus: black outportions of the panoramic projection, the blacked out portions of thepanoramic projection including redundant imaging data.

In yet another variant, the computer-readable instructions are furtherconfigured to, when executed by the processor apparatus: insert metadatainformation into the blacked out portions of the panoramic projection.

In a twelfth aspect, an integrated circuit (IC) apparatus is disclosed.In one embodiment, the IC apparatus is configured to perform at least aportion of the aforementioned methodologies described herein.

In a variant, the integrated circuit apparatus is configured to encodeSSP imaging data and/or RSP imaging data into an extant imaging format.

In another variant, the integrated circuit apparatus is configured todecode SSP imaging data and/or RSP imaging data from an extant imagingformat.

In a thirteenth aspect, a computing device is disclosed. In oneembodiment, the computing device is configured to perform at least aportion of the aforementioned methodologies described herein.

In one variant, the computing device is configured to encode and/ordecode SSP imaging data and/or RSP imaging data to/from an extantimaging format.

In a variant, the computing device is configured to display the decodedSSP imaging data and/or RSP imaging data.

In another variant, the imaging format includes an RSP-3×2 or RSP-2×3imaging format.

Other features and advantages of the present disclosure will immediatelybe recognized by persons of ordinary skill in the art with reference tothe attached drawings and detailed description of exemplaryimplementations as given below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a graphical representation of an exemplary Segmented SphereProjection in a 6×1 arrangement (SSP-6×1), useful in describing theprinciples of the present disclosure.

FIG. 1B is a graphical representation of an exemplary Rotated SphereProjection in a 6×1 arrangement (RSP-6×1), useful in describing theprinciples of the present disclosure.

FIG. 1C is a graphical representation of an exemplary Segmented SphereProjection in a 1×6 arrangement (SSP-1×6) for use with existing codecs,useful in describing the principles of the present disclosure.

FIG. 2 is a graphical illustration depicting viewport change whenviewing panoramic content in accordance with the principles of thepresent disclosure.

FIG. 3A is a graphical representation of a first exemplary embodiment ofa frame packing arrangement for mono captured images, useful indescribing the principles of the present disclosure.

FIG. 3B is a graphical representation of a second exemplary embodimentof a frame packing arrangement for mono captured images, useful indescribing the principles of the present disclosure.

FIG. 3C is a graphical representation of a third exemplary embodiment ofa frame packing arrangement for mono captured images, useful indescribing the principles of the present disclosure.

FIG. 3D is a graphical representation of a fourth exemplary embodimentof a frame packing arrangement for mono captured images, useful indescribing the principles of the present disclosure.

FIG. 3E is a graphical representation of a fifth exemplary embodiment ofa frame packing arrangement for mono captured images, useful indescribing the principles of the present disclosure.

FIG. 4A is a graphical representation of a first exemplary embodiment ofa frame packing arrangement for stereo captured images, useful indescribing the principles of the present disclosure.

FIG. 4B is a graphical representation of a second exemplary embodimentof a frame packing arrangement for stereo captured images, useful indescribing the principles of the present disclosure.

FIG. 4C is a graphical representation of a third exemplary embodiment ofa frame packing arrangement for stereo captured images, useful indescribing the principles of the present disclosure.

FIG. 4D is a graphical representation of a fourth exemplary embodimentof a frame packing arrangement for stereo captured images, useful indescribing the principles of the present disclosure.

FIG. 4E is a graphical representation of a fifth exemplary embodiment ofa frame packing arrangement for stereo captured images, useful indescribing the principles of the present disclosure.

FIG. 4F is a graphical representation of a sixth exemplary embodiment ofa frame packing arrangement for stereo captured images, useful indescribing the principles of the present disclosure.

FIG. 5A is a graphical representation of a first cropped region for usewith, for example, RSP-3×2 frame packing arrangements, useful indescribing the principles of the present disclosure.

FIG. 5B is a graphical representation of a second cropped region for usewith, for example, RSP-3×2 frame packing arrangements, useful indescribing the principles of the present disclosure.

FIG. 5C is a graphical representation of a spherical coordinate system,useful in describing the principles of the present disclosure.

FIG. 5D is a graphical representation of an exemplary RSP-3×2 framepacking arrangement using the first and second cropped regions of FIGS.4A and 4B, useful in describing the principles of the presentdisclosure.

FIG. 5E is a graphical representation of a depiction of the RSP-3×2frame packing arrangement of, for example, FIG. 4D, useful in describingthe principles of the present disclosure.

FIG. 5F is a graphical representation of one exemplary RSP-2×3 framepacking arrangement using the first and second cropped regions of FIGS.4A and 4B, useful in describing the principles of the presentdisclosure.

FIG. 5G is a graphical representation of another exemplary RSP-2×3 framepacking arrangement using the first and second cropped regions of FIGS.4A and 4B, useful in describing the principles of the presentdisclosure.

FIG. 5H is a graphical representation of one exemplary RSP 3×2 framepacking arrangement illustrating redundant imaging data being blackedout from the frame, useful in describing the principles of the presentdisclosure.

FIG. 5I is a graphical representation of another exemplary RSP 3×2 framepacking arrangement illustrating redundant imaging data being blackedout from the frame, useful in describing the principles of the presentdisclosure.

FIG. 5J is a graphical representation of yet another exemplary RSP 3×2frame packing arrangement illustrating redundant imaging data beingblacked out from the frame, useful in describing the principles of thepresent disclosure.

FIG. 5K is a graphical representation of an exemplary frame packingarrangement comprising equirectangular projection (ERP), useful indescribing the principles of the present disclosure.

FIG. 5L is a graphical representation of an exemplary frame packingarrangement comprising equal area projection (EAP), useful in describingthe principles of the present disclosure.

FIG. 6A is a logical flow diagram illustrating an exemplary embodimentfor creating a target panoramic projection, useful in describing theprinciples of the present disclosure.

FIG. 6B is a logical flow diagram illustrating an exemplary embodimentfor decoding and displaying a target panoramic projection, useful indescribing the principles of the present disclosure.

FIG. 7 is a block diagram illustrating an exemplary system for theencoding and decoding of a target panoramic projection, useful indescribing the principles of the present disclosure.

FIG. 8A is a graphical representation of an exemplary RSP 3×2 framepacking arrangement, useful in describing the principles of the presentdisclosure.

FIG. 8B is a graphical representation of the exemplary RSP frame packingarrangement of FIG. 8A in which the bottom facet has been downsampledvertically, useful in describing the principles of the presentdisclosure.

FIG. 8C is a graphical representation of the exemplary RSP frame packingarrangement of FIG. 8A in which the bottom facet has been downsampledvertically and horizontally and rearranged, useful in describing theprinciples of the present disclosure.

FIG. 8D is a graphical representation of the exemplary RSP frame packingarrangement of FIG. 8A in which portions of the bottom facet have beendownsampled, useful in describing the principles of the presentdisclosure.

FIG. 8E is a graphical representation of the exemplary RSP frame packingarrangement of FIG. 8D in which the bottom facet has been rearranged,useful in describing the principles of the present disclosure.

FIG. 9A is a logical flow diagram illustrating an exemplary embodimentfor adjusting the quantization parameter of one or both of the two imagefacets of a rotated sphere projection, useful in describing theprinciples of the present disclosure.

FIG. 9B is a logical flow diagram illustrating an exemplary embodimentfor transmitting panoramic content in a rotated sphere projection thathas been optimized for streaming applications, useful in describing theprinciples of the present disclosure.

FIG. 10 is a block diagram of an exemplary implementation of a computingdevice, useful in encoding and/or decoding the exemplary frame packingarrangements as described herein.

All Figures disclosed herein are © Copyright 2017 GoPro, Inc. All rightsreserved.

DETAILED DESCRIPTION

Implementations of the present technology will now be described indetail with reference to the drawings, which are provided asillustrative examples and species of broader genus' so as to enablethose skilled in the art to practice the technology. Notably, thefigures and examples below are not meant to limit the scope of thepresent disclosure to any single implementation or implementations, butother implementations are possible by way of interchange of,substitution of, or combination with some or all of the described orillustrated elements. Wherever convenient, the same reference numberswill be used throughout the drawings to refer to same or like parts.

Moreover, while implementations described herein are primarily discussedin the context of frame packing arrangements for so-called SegmentedSphere Projections (SSP) and Rotated Sphere Projections (RSP) such asthat described in co-owned and co-pending U.S. Provisional PatentApplication Ser. No. 62/446,297 filed Jan. 13, 2017 and entitled“Methods and Apparatus for Rotated Sphere Projections”, U.S. ProvisionalPatent Application Ser. No. 62/465,678 filed Mar. 1, 2017 and entitled“Methods and Apparatus for Providing a Frame Packing Arrangement forPanoramic Content”, and U.S. Provisional Patent Application Ser. No.62/473,952 filed Mar. 20, 2017 and entitled “Methods and Apparatus forProviding a Frame Packing Arrangement for Panoramic Content”, thecontents of each of the foregoing incorporated supra, it is readilyappreciated that the principles described herein can be equally appliedto other projection formats. For example, the frame packing arrangementsdescribed herein may be readily applied to other types of projectionsfor panoramic content (e.g., 360°) that may have an asymmetric facelayout including, for example, those projections and techniquesdescribed in co-owned and co-pending U.S. patent application Ser. No.15/289,851 filed Oct. 10, 2016 and entitled “Apparatus and Methods forthe Optimal Stitch Zone Calculation of a Generated Projection of aSpherical Image”, the contents of which were incorporated supra.

Additionally, while primarily discussed in the context of encoding anddecoding of 4K image resolutions having a variety of aspect ratios(e.g., 4,096×2,048 pixels), it would be readily appreciated by one ofordinary skill given the contents of the present disclosure that theprinciples described herein may be readily applied to other imagingformats and aspect ratios. For example, the principles described hereinmay be readily applied to various display resolutions including, forexample, high definition (HD) variants, 4K variants, 8K variants, andthe like, and at a variety of differing display aspect ratios (e.g., 4:3and 16:9 aspect ratios, etc.) for various ones of these aforementioneddisplay resolutions.

While examples of the present disclosure are presented within thecontext of static photography, artisans of ordinary skill in the relatedarts will readily appreciate that the various principles describedherein may be equally applied to a wide range of imaging applications,including e.g., video capture, video rendering, virtual reality (VR),augmented reality (AR) and the like. For example, a panoramic image canbe generated from a video capture while rotating a camera (e.g.,stitching together the individual frames in time as different fields ofview (FOV)). Similarly, source images may be dynamically stitchedtogether during a video playback (e.g., for virtual reality (VR),augmented reality (AR) applications, mixed reality, augmentedvirtuality, and/or other hybridized realities).

These and other variations would be readily apparent to one of ordinaryskill given the contents of the present disclosure.

Exemplary Frame Packing Arrangement for Projections—

FIG. 1A illustrates an exemplary frame of SSP imaging data 100. SSP mayhave four (4) faces 102, 104, 106, 108 which may be directly derivedfrom an Equirectangular Projection (ERP), while the faces 110, 120representing the poles of the image (e.g., top 110 and bottom 120images) may be different from the aforementioned ERP. As illustrated inFIG. 1A, the top 110 and bottom 120 of the images are each depicted in afish eye projection. FIG. 1B illustrates an exemplary frame of RSPimaging data 150. As used herein, the term rotated sphere projection orRSP includes a projection that uses some portions of the image from anoriginal non-rotated image (using any projection), while using otherportions of the original non-rotated image after applying a sphericalrotation and re-projection onto the original or any other projection.For example, and similar to the depicted SSP image, RSP may also havefour (4) faces 152, 154, 156, 158 which may be directly derived from anERP that runs along the equator. However, unlike the SSP image 100 ofFIG. 1A, the top 160 and bottom 170 images illustrated in FIG. 1B arederived by altering the ERP projection such that the equator in the four(4) faces has been rotated such that it now runs along the meridian ofthe spherical image. Accordingly, by moving the north and south (top 160and bottom 170) polar regions of the image to the new “equator”(meridian) of the altered ERP projection, the top 160 and bottom 170images can be sampled and reproduced with minimal distortion (ascompared with, for example, the aforementioned SSP frame).

The aforementioned SSP and RSP projections have a number of advantagesover other prior techniques including possessing a better compressionefficiency when encoding the image. For example, because both the SSPand RSP projections may be closer geometrically to the capturedspherical image, they can be reproduced with a lesser amount ofgeometric distortion and with fewer face discontinuities as comparedwith other geometric projections (e.g., cubic, octahedron, icosahedron,and the like). Additionally, and as discussed in additional detailinfra, they often perform better on extant codec tests than at leastsome of the aforementioned other geometric projections. Additionally,these SSP and RSP projections may be represented with, for example, a25% reduction in the number of coded pixels and hence, may betransmitted at a higher resolution (or alternatively at a reducedbandwidth/bit-rate for a given image quality and/or other bandwidthconsiderations) by using, for example, extant codec architectures ascompared with at least some of the aforementioned other geometricprojections.

The SSP imaging data 100 of FIG. 1A and RSP imaging data 150 of FIG. 1Bas illustrated possesses an aspect ratio of 6:1. Accordingly, in thecontext of an exemplary 4K frame of imaging data (i.e., 4,096×2,048pixels), the SSP imaging data 100 and RSP imaging data 150 may have anoverall pixel size of 6,144×1,024 pixels. This wide aspect ratio for theSSP imaging data 100 and RSP imaging data 150 have a number of readilyidentifiable deficiencies. First, and in the exemplary context of 4Kframes of imaging data, most (if not all) already deployed codecs aredesigned to handle at most a single frame of 4K data at a time.Accordingly, many existing 4K codec line buffers are designed such thatthe pixel width of the image does not exceed 4,096 pixels. Additionally,with the aforementioned 4K codec line buffer limitationsnotwithstanding, the wide 6:1 aspect ratio is often not well suited forconsumption in a “raw” format on, for example, 16:9 aspect ratiomonitors, television sets, and smartphone screens. As a result, a viewerof the wide 6:1 aspect ratio content may not be able to view the contentin high resolution as the width of modern display devices often becomesa limiting factor. Additionally, wider aspect ratios (such as theaforementioned 6:1 aspect ratio) may not lend well to some parallelprocessing tools. For example, this aforementioned 6:1 aspect ratio maynot be suitable for so-called slices in H.264/AVC. Many commonmulti-slice implementations of H.264 encoders may split a 6:1 aspectratio frame into thin rectangular slices, which may be inefficient froma compression perspective. As a brief aside, slices may tend to be moreefficient for the purposes of encoding when they are more square inaspect ratio.

Referring now to FIG. 1C, one such SSP imaging data 180 approach forovercoming, inter alia, the aforementioned 4K line buffer limitations isillustrated. As can be seen, the SSP data 180 has been rotated by 90° sothat it now resides in a 1:6 aspect ratio (e.g., 1,024×6,144 pixels).While the line buffer limitations have been addressed with the SSPimaging data 180 of FIG. 1C, this 1:6 aspect ratio may look even worsewith modern display devices (when looking at the raw imaging data). Forsome modern displays (such as e.g., smart phones), this may be asuitable approach when the phone is held in portrait mode (as opposed tobeing held in landscape mode). However, the content will appear rotatedand may not be aesthetically pleasing when looking at the raw imagingdata. Additionally, image post processing methodologies for the imagingformat for FIG. 1C may also need to apply a rotation to the images,which may result in additional processing resources being required at,for example, the image encoder and/or image decoder. The substitution ofRSP imaging data in a 1:6 aspect ratio may also possess similardisadvantages.

Additionally, the SSP imaging data 180 depicted in FIG. 1C may haveproblems in terms of compression efficiency for extant codecs. Forexample, many existing codecs are designed to handle content in a wideraspect ratio (with more horizontal as opposed to vertical motion (e.g.,camera pans)). Accordingly, this tall and rotated orientation asillustrated in FIG. 1C may result in sub-optimal codec performance.Additionally, using a 1:6 aspect ratio image may mean that the codecpipeline may be sub-optimally used (which in the context of 4K displayresolutions was designed to handle 4K line buffers). This may result inworse performance as there are now more vertical lines to process (e.g.,worst case processing of these images may be equivalent to processing a4,096×6,144 image).

As a brief aside, FIG. 2 illustrates viewport change when viewingpanoramic content (such as the panoramic content described withreference to FIGS. 1A-1C as well as the panoramic content described inthe various frame packing arrangements discussed infra), in accordancewith one implementation. In some implementations a user may viewpanoramic content using a virtual reality (VR) headset, 202 in FIG. 2.Headset 202 may include a sensor component configured to provideinformation related to orientation and/or motion of headset 202. In someimplementations, the sensor component may include an accelerometer, atilt sensor, a compass, a heading sensor, a gyroscope, and/or othersensors.

When headset 202 is pointed in a given direction, for example, as shownin panel 200 in FIG. 2, the viewport associated with the position ofheadset 202 may be denoted by area 212 within the panoramic image frame210. As used herein the terms “viewport” and/or “view area” may be usedto describe a portion of a view field that may be used for viewingpanoramic content that may be characterized by a content view field(e.g., shown by frame 210 in FIG. 2). When panoramic content ispresented on a two dimensional display device, the viewport may denote atwo dimensional area (e.g., area 212) within the 2-dimensionalprojection of the acquired panoramic content (e.g., frame 210).

When providing a portion of the panoramic content (e.g., a viewport 212)to a client device, a portion of the content corresponding to thepresent viewport may be encoded, transmitted, and/or decoded to reduceload on a content server, transmission resource (e.g., bandwidth,energy) utilization, and/or client device decoder load. Viewport changesmay necessitate content bitstream adjustment. By way of an illustration,as the head of a user moves from configuration 200 to configuration 220in FIG. 2, the viewport may change, e.g., from area 212 to area 222within the panoramic content frame 210. Accordingly, the contentproviding entity (e.g., content server) may need to transition fromproviding bitstream from content within the area 212 to contentassociated with the area 222.

Mono Configuration Frame Packing—

Referring now to FIGS. 3A-3E, exemplary implementations of frame packingarrangements for mono captured images are shown and described in detail.In the context of capturing, the term “mono” as used herein may refer tothe fact that a significant portion of a captured 360° image is capturedwithout significant overlap, hence perception of depth is notreproduced. In the context of streaming and playback, the term “mono”may also refer to the fact that both the left and right eyes see thesame images during content consumption. As discussed elsewhere herein,panoramic images (e.g., spherical images) are typically obtained bycapturing multiple images with overlapping fields of view from differentcameras and combining (“stitching”) these images together in order toprovide, for example, a two-dimensional projection for use with moderndisplay devices. Additionally, panoramic images may be obtained bycapturing multiple images with overlapping fields of view with a singlecamera that is rotated about a two-dimensional axis, orthree-dimensional axes. These stitched images may be captured using theaforementioned mono image capturing techniques, or alternatively may becaptured using stereoscopy techniques for creating the illusion of depthin an image by means of capturing a given FOV with two (or more) imagecapturing devices, or by applying an offset in order to create twoseparate images from a single captured image. Stereo configuration framepacking will be described in subsequent detail herein with respect toFIGS. 4A-4F.

FIG. 3A illustrates one such exemplary frame packing configuration 300for the packing of a frame of, for example, SSP and/or RSP imaging data.In some implementations, the ERP portion of the SSP or RSP imaging datawill be stored in a frame packing width 302 and will have a framepacking depth 304. The circular faces of the SSP imaging data (e.g.,faces 110, 120 in FIG. 1A), or the RSP imaging data (e.g., faces 160,170 in FIG. 1B) will be segmented and stored in the frame packing width302 and will have a frame packing depth 306. Additionally, the circularfaces of the SSP imaging data, or the RSP imaging data are oriented suchthat more of the pixels associated with these circular faces are closerto the pixels of, for example, the ERP portion, thereby making them moreefficient from a caching perspective in some implementations.

In the exemplary context of the aforementioned 4K imaging resolution,the frame packing width 302 may have a pixel width of 4,096 pixels, theframe packing depth 304 may have a pixel depth of 1,024 pixels, and theframe packing depth 306 may have a pixel depth of 512 pixels.Accordingly, the aspect ratio for this particular configuration will be8:3 (e.g., an SSP 8:3 frame packing format or RSP 8:3 frame packingformat). Herein lies one salient advantage of the frame packingconfiguration 300 depicted in FIG. 3A, namely the overall size of theframe packing configuration may have a 25% reduction in the pixelscontained within a frame as compared with a native 4K resolution frame(i.e., 4,096 pixels×1,536 pixels as compared with 4,096 pixels×2,048pixels), thereby resulting in increased processing overhead savings aswell as reduced bandwidth considerations during transmission/reception.

Referring again to the circular faces depicted in FIG. 3A, and in theexemplary context of a panoramic image captured using, for example, atwo camera image capturing device arranged in a Janus configuration(e.g., in a back-to-back configuration), segmented portion A may referto a portion of the image that is looking upwards and captured using aforward facing camera, while segmented portion B may refer to a portionof the image that is also looking upwards and captured using a rearwardfacing camera. Similarly, segmented portion C may refer to a portion ofthe image that is looking downwards and captured using a forward facingcamera, while segmented portion D may refer to a portion of the imagethat is also looking downwards and captured using a rearward facingcamera. While the discussion of a two camera image capturing devicearranged in a Janus configuration, it will be readily appreciated by oneof ordinary skill given the contents of the present disclosure, thatother suitable image capturing devices (such as 6-axis image capturingdevices, such as that described in co-owned and co-pending U.S. patentapplication Ser. No. 15/414,403 filed Jan. 24, 2017 and entitled“Systems and Methods for Compressing Video Content”, the contents ofwhich are incorporated herein by reference in its entirety), may bereadily applied to the frame packing arrangements as disclosed herein.

In some implementations, the depicted dark region 308 may contain nulldata, thereby enabling reduced processing overhead for, inter alia, theencoding/decoding of frame packing configuration 300. Additionally, thenull data may be enabled for reduced transmission bandwidth and lowerbit rates. In alternative implementations, some or all of the depicteddark region 308 may include metadata information that may, for example,be utilized for the stitching of the captured panoramic for display on acomputing device (such as computing device 1000 depicted in FIG. 10).Additionally, or alternatively, in some implementations, the depicteddark region 308 may include information that enables a decoder todetermine the particular encoding format configuration chosen such as,for example, by determining which frame packing configuration 300, 320(FIG. 3B), 340 (FIG. 3C) and the like has been utilized for the encodingprocess.

In some implementations, the depicted dark region 308 may includeadditional information such as that disclosed in co-owned and co-pendingU.S. patent application Ser. No. 15/406,175 filed Jan. 13, 2017 entitled“Apparatus and Methods for the Storage of Overlapping Regions of ImagingData for the Generation of Optimized Stitched Images”, which is alsoincorporated herein by reference in its entirety. In one or moreimplementations, a decoder may utilize information contained within thedepicted dark region 308 for the decoding/rendering of the frame, while,for example, legacy decoders may simply discard the data containedwithin dark region 308.

Referring now to FIG. 3B, a second exemplary frame packing configuration320 is illustrated. The frame packing configuration 320 may includeseveral of the advantages described herein with reference to FIG. 3A;however, frame packing configuration 320 differs from that depicted inFIG. 3A in that the circular faces A, B, C, D of, for example, the SSPimaging data (e.g., faces 110, 120 in FIG. 1A), or the RSP imaging data(e.g., faces 160, 170 in FIG. 1B) may be inverted.

Referring now to FIG. 3C, yet another exemplary frame packingconfiguration 340 is illustrated. Similar to the discussion of FIG. 3Bsupra, the frame packing configuration 340 of FIG. 3C may includeseveral of the advantages described herein with reference to FIG. 3A;however, frame packing configuration 340 differs from that depicted inFIG. 3A in that the circular faces A, B, C, D of, for example, the SSPimaging data (e.g., faces 110, 120 in FIG. 1A), or the RSP imaging data(e.g., faces 160, 170 in FIG. 1B), may be alternately inverted.

For example, image portion A may be arranged similar with respect toimage portion A within FIG. 3B, while image portion B may be arrangedsimilar with respect to image portion B within FIG. 3A. Similarly, imageportion C may be arranged similar with respect to image portion C withinFIG. 3B, while image portion D may be arranged similar with respect toimage portion D within FIG. 3A. In one or more implementations, thealternately inverted imaging portions A, B, C, D as depicted in FIG. 3Cmay be configured to enable improved compression efficiencies using, forexample, existing compression methodologies. For example, an objectdetected within imaging portion A may travel predictably (e.g., in asame or similar motion trajectory) when this object may be depicted inimaging portion B in subsequent frame(s). Accordingly, the frame packingconfiguration 340 depicted in FIG. 3C may compress more efficiently ascompared with for example, frame packing configuration 300 (FIG. 3A) andframe packing configuration 320 (FIG. 3B). Additionally, the framepacking arrangement of FIG. 3C may visually be the most natural,especially when the image capture device may be in motion (e.g., duringdirect viewing of the captured image(s)).

Each of the frame packing configurations 300, 320, 340 depicted in FIGS.3A-3C may advantageously preserve line buffer requirements for existingcodecs while simultaneously optimally using the line buffers of theseexisting codecs as compared with, for example, the depicted framepacking configurations depicted within FIGS. 1A-1C. Additionally, theaspect ratio of these frame packing configurations 300, 320, 340 (e.g.,an 8:3 aspect ratio) may also facilitate multi-slice and multi-tileencoding techniques of the type known by one of ordinary skill in theencoding arts, as the aspect ratio is closer to a 1:1 aspect ratio ascompared with, for example, the frame packing configurations illustratedin FIGS. 1A-1C. As a result, one may pack the same number of slices andtiles, which are typically squarer in shape, thereby achieving a highercompression efficiency during encoding as compared with, for example,the frame packing configurations illustrated in FIGS. 1A-1C.

FIG. 3D illustrates yet another exemplary frame packing configuration360 for the packing of a frame of, for example, SSP and/or RSP imagingdata. In the illustrated frame packing configuration 360, the frame haseffectively been rotated by 90° in a counter-clockwise rotation ascompared with, for example, the frame packing configuration 300illustrated in FIG. 3A. The exemplary frame packing configuration 360enables one to reduce the effective pixel width of the frame, at theexpense of increased height.

In some implementations, such as in applications in which where thereare line buffer processing limitations, the exemplary frame packingconfiguration 360 may have advantages, including being more well suitedfor particular types of line buffers. Additionally, the circular facesof the SSP imaging data, or the RSP imaging data are oriented such thatmore of the pixels associated with these circular faces are closer tothe pixels of, for example, the ERP portion, thereby making them moreefficient from a caching perspective in some implementations. Asillustrated in FIG. 3E, the exemplary frame packing configuration 380 issimilar to that shown in FIG. 3D; however, the circular faces of the SSPimaging data, or the RSP imaging data are oriented such that fewer ofthe pixels associated with these circular faces are closer to the pixelsof, for example, the ERP portion, which may be advantageous in someimplementations.

In the exemplary context of the aforementioned 4K imaging resolution,the frame packing width for the frame packing arrangements 360, 380,shown in respective FIGS. 3D and 3E will have a pixel width that isequivalent to the combined pixel width of pixel width 304 of 1,024pixels and pixel width 306 of 512 pixels (i.e., a combined pixel widthof 1,536 pixels). Moreover, the pixel depth 302, may have a pixel depthof 4,096 pixels. Accordingly, the aspect ratio for these particularconfigurations will be 3:8 (e.g., an SSP 3:8 frame packing format or RSP3:8 frame packing format).

Stereo Configuration Frame Packing—

Referring now to FIGS. 4A-4F, exemplary implementations of frame packingarrangement for stereo captured images are shown and described indetail. As a brief aside, stereo imaging data may be used to create asense of depth when viewing the stereo imaging data on, for example, adisplay device suitable for use in viewing stereo imaging data. One suchexemplary stereo display device includes a head mounted display devicethat is configured to display right eye imaging data into the right eyeof a user, while displaying left eye imaging data into the left eye of auser. This stereo imaging data may present two offset images to the leftand right eye of user, respectively. These separate two-dimensionalimages are then combined in the brain of the viewer to give the viewerof this data the perception of three-dimensional depth. These stereoimages are typically captured using an image capturing device thatincludes a plurality of cameras. Exemplary image capture devices for thecapture of stereo imaging data include the Facebook® “Surround 360”camera rig and the Jaunt® “Jaunt One” camera rig. In order to capturepanoramic images (e.g., 360° FOV and/or other panoramic FOV), thecameras located on the image capturing device capture stereo imagesalong the equatorial plane of the image capturing device while capturingmono images along the poles (top and/or bottom) of the device.Additionally, the image capture devices may capture stereo imageryoffset from the given equatorial plane (e.g., at 30°, 45°, 60°, etc.).

FIG. 4A depicts one exemplary frame packing arrangement 400 for thepacking of stereo imaging data. For example, the Google® JUMP systemarranges captured stereo imaging data in a format similar to thatdepicted in FIG. 4A. The frame packing arrangement 400 illustratedincludes a pixel width 402 and a pixel depth 404 into which two separateERP images are stacked one on top of the other (e.g., left imaging dataon the top and right imaging data on the bottom, although it would beappreciated that the arrangement of this left and right imaging data maybe reversed in alternative arrangements, including in the alternativeframe packing arrangements depicted in FIGS. 4B-4F).

FIG. 4B depicts an alternative frame packing arrangement 420 in whichthe left and right ERP imaging data also includes left and right top andbottom view imaging data. As illustrated, the frame packing arrangement420 may possess a frame pixel width 402 with a pixel depth 406containing, for example, left image ERP imaging data, pixel depth 408containing left image fish eye projection or ERP projection top andbottom view data, pixel depth 410 containing right image ERP imagingdata, and pixel depth 412 containing right fish eye projection or ERPprojection top and bottom view data. By stacking the left and rightimaging data using, for example, the 8:3 SSP or 8:3 RSP frame packingarrangement of FIG. 3B, the resultant frame packing arrangement 420 ofFIG. 4B results in a 4:3 aspect ratio.

A 4:3 aspect ratio for frame packing arrangement 420 is advantageous asthis stereo imaging data may be encoded/decoded using extant codecs thatare specifically tailored to handle 4:3 aspect ratio images. However,the frame packing arrangement 420 may be sub-optimal for the purposesof, for example, compression efficiency during the encoding process asframe packing arrangement 420 doesn't employ any continuity along theequatorial axis of the ERP projection data (e.g., left and right imagingdata), nor along the top and bottom images (e.g., L_(A), L_(B), L_(C),L_(D), R_(A), R_(B), R_(C), R_(D)). In addition, and similar to thediscussion with regards to FIG. 3A, the frame packing arrangement 420 ofFIG. 4B also includes depicted dark regions 414.

In some implementations, the depicted dark regions 414 may contain nulldata, thereby enabling reduced processing overhead for, inter alia, theencoding/decoding of frame packing configuration 420. Additionally, thenull data may be enabled for reduced transmission bandwidth and lowerbit rates. In alternative implementations, some or all of the depicteddark regions 414 may include metadata information that may, for example,be utilized for the stitching of the captured panoramic for display on acomputing device. Additionally, or alternatively, in someimplementations, the depicted dark regions 414 may include informationthat enables a decoder to determine the particular encoding formatconfiguration chosen such as, for example, determining which framepacking configuration 420, 440 (FIG. 4C), 450 (FIG. 4D), 460 (FIG. 4E),470 (FIG. 4F) has been utilized for the encoding process. In someimplementations, the depicted dark regions 414 may include additionalinformation such as that disclosed in co-owned and co-pending U.S.patent application Ser. No. 15/406,175 filed Jan. 13, 2017 entitled“Apparatus and Methods for the Storage of Overlapping Regions of ImagingData for the Generation of Optimized Stitched Images”, incorporatedsupra. In one or more implementations, a decoder may utilize informationcontained within the depicted dark regions 414 for thedecoding/rendering of the frame, while, for example, legacy decoders maysimply discard the data contained within dark regions 414.

FIG. 4C illustrates one such alternative frame packing arrangement 440that addresses some of the aforementioned continuity deficiencies of theframe packing arrangement of FIG. 4B. Specifically, frame packingarrangement 440 differs from that shown in FIG. 4B as the right imagingdata (e.g., ERP imaging data) within pixel depth 410 has been: (1)inverted; and (2) reversed. By inverting and reversing the right imagingdata, better continuity is achieved as a result of the fact that, forexample, the equatorial part of the right image data has been placedadjacent to the equatorial part of the left image data. In someimplementations, it may be desirable to either invert or reverse one ofthe left or right imaging data as opposed to inverting and reversing oneof the left or right imaging data. This additional continuity may resultin improved compression efficiency during the encoding process of thisstereo imaging data; however, the encoding of the top images (e.g.,L_(A), L_(B), R_(A), R_(B)) and bottom images (e.g., L_(C), L_(D),R_(C), R_(D)) may be sub-optimal for the purposes of improving uponcompression efficiency during the encoding process.

FIG. 4D illustrates one such alternative frame packing arrangement 450that continues to improve upon some of the aforementioned continuitydeficiencies of the arrangement of FIG. 4C. Specifically, in framepacking arrangement 450, the top images (e.g., L_(A), L_(B), R_(A),R_(B)) and bottom images (e.g., L_(C), L_(D), R_(C), R_(D)) have beenplaced in close proximity with respect to one another, improving uponcontinuity and improving upon the corresponding compression efficiencyof encoding these top and bottom images.

FIG. 4E illustrates yet another alternative frame packing arrangement460 for the packing of stereo imaging data. Specifically, the framepacking arrangement 460 of FIG. 4E is similar to the frame packingarrangement 450 of FIG. 4D; however, the packing of the top and bottomimaging data differs from that illustrated in FIG. 4D. Specifically, fora given left image portion of the top and bottom images (e.g., imageportion L_(A)), a corresponding right image portion of the top andbottom image is placed adjacent to the left image portion (e.g., imageportion R_(A) for image portion L_(A)). While the frame packingarrangement 460 of FIG. 4E does not have as good as continuity, andassociated improved compression efficiencies with regards to the framepacking arrangement 450 of FIG. 4D, the frame packing arrangement 460 ofFIG. 4E does have its advantages.

As a brief aside, stereo image data typically employs improved offset(and associated perception of depth) along a given equatorial plane. Asa viewer of the content fixes their gaze away from the given equatorialplane (e.g., looks up, looks down, etc.), the offset between thecorresponding left and right images may diminish. In particular, stereoimaging data tends to approach the look of mono imaging data at thepoles, while maintaining the perception of depth along the givenequatorial plane. In particular, and in some implementations, if stereoimagery was provided at the poles (i.e., away from the given equatorialplane) and the viewer turned their head by, for example, 180°, the leftand right eye imagery may become swapped, resulting in, inter alia,nausea and fatigue for the viewer. Additionally, if a viewer of thestereo imagery looked at an off-kilter viewing angle (i.e., where oneeye may be focused farther above/below the given equatorial plane thanthe other eye), this may result in, inter alia, nausea and fatigue aswell for the user. In some implementations, this may be the primaryreasoning for why extant stereo VR camera rig manufacturers do not evenbother collecting stereo imagery at, for example, the top and bottom(i.e., poles) orthogonal with the given equatorial plane. As a result,in some implementations, the respective views of the left/right imagestend to have less and less offset (e.g., become near identical) as aviewer fixes his gaze away from the equatorial plane.

Accordingly, the frame packing arrangement 460 of FIG. 4E may lenditself to simply obviating this nearly redundant or redundant data whenlooking towards the poles of the captured image(s) by simplydisregarding the imaging data located in pixel depth 412 (oralternatively, pixel depth 408). For example, a decoder may simplydisregard this near redundant data or redundant data when decoding framepacking arrangement 460 of FIG. 4E. In some implementation, an encodermay simply choose not to encode this near redundant data (e.g.,discarding this data during the encoding process) in order toeffectively gain the frame packing arrangement 470 depicted in FIG. 4F.Hence, here lies one salient advantage for the frame packingarrangements 460, 470 illustrated in FIGS. 4E and 4F. Namely, theability to use this optional imaging data “on the fly”. For example, ifthere is a need to present full stereo imagery (depth) at the poles(e.g., as a result of having sufficient available bitrate), a signal maybe sent to the encoder that is indicative that frame packing arrangement460 should be utilized. If however, the bitrate requirement drops and/orthe application needs to present mono data to a viewer at the poles, itmay either encode frame packing arrangement 470 or may simply discardpixel depth 412 (e.g., crop off the bottom part of the image) of framepacking arrangement 460.

FIG. 4F illustrates an exemplary frame packing arrangement 470 that mayhave a 37.5% lower amount of pixels for a given frame of data than, forexample, the frame packing arrangement 400 shown in FIG. 4A. Forexample, in the exemplary context of 4K resolution imaging data, framepacking arrangement 400 of FIG. 4A may have a pixel width of 4,096pixels and a pixel depth of 4,096 pixels (e.g., two 4,096×2,048 pixelimages per frame). However, frame packing arrangement 470 may have asimilar pixel width (e.g., 4,096 pixels), but a significantly reducedpixel depth (e.g., 2,560 pixels) resulting in an 8:5 aspect ratio.Accordingly, by utilizing the frame packing arrangement 470 of FIG. 4F,one may only require, for example, 25% more pixels for 4K stereo imagingdata as compared with non-stereo (mono) 4K imaging data, while stillproviding for stereo image/video data along the given equatorial planeof the image capturing device, which may be a better tradeoff undergiven bandwidth/complexity and latency constraints.

Additionally, additional compression efficiency using, for example,existing codecs can be increased if a motion search window is largeenough to traverse across imaging facets (e.g., L_(A) and L_(B) and/orL_(C) and L_(D), as but two of many examples). Moreover, codecs may beutilized to motion compensate across the imaging facets (e.g.,equatorial (e.g., ERP), top, and bottom) in, for example, three separateencoding loops, thereby improving upon the resultant compressionefficiency for the encoding of these stereo images.

Alternative Two Facet Frame Packing Arrangements—

As a brief aside, and generally speaking, in order to obtain bettercoding efficiency utilizing, for example, existing codecs, it is oftendesirable to have as few seams (or discontinuities) as possible. Havinga lower number of seams may result in a better subjective quality forthe image as, for example, the seam surface area gets smaller ascompared with the total area on the captured sphere (or capturedpanoramic image). For example, and referring to the aforementioned 6:1and 1:6 aspect ratio frame packing arrangements shown in FIGS. 1A-1C ascompared with the 8:3 and 3:8 aspect ratio frame packing arrangementsshown in FIGS. 3A-3E; the 8:3 and 3:8 will have the advantages asdiscussed previously herein, while only increasing the level ofdiscontinuity by approximately 3%. Accordingly, it may be desirable todesign a projection/frame packing arrangement that achieve many, if notall, the benefits of the 8:3 and 3:8 aspect ratio frame packingarrangements shown in FIGS. 3A-3E, while further reducing the amount ofdiscontinuity associated with the image.

Accordingly, a desired projection/frame packing arrangement may have:(1) a lower number of discontinuities, so that codecs can perform moreoptimally while reducing the difficulties associated with dealing withseam issues; (2) possessing an arrangement that is closer to a 16:9aspect ratio such that line buffers get optimally used and providingimproved performance for parallel processing tools on existing codecs(e.g., slices in H.264 where square slices tend to work more optimally);and (3) a layout that is more ready to render. For example, andreferring to item (3), for the purposes of live streaming video contentor for previewing-type applications, it may be desirable not to have tore-project in order to extract a viewport from the panoramic content,but rather to perform, for example, a relatively simple pixel crop fromthe originally projected video content. In other words, performing apixel crop may be more advantageous from, for example, a processingoverhead/memory resource allocation, etc. point of view as compared withhaving to perform, for example, a re-projection operation, imagestitching operations and the like.

Referring now to FIGS. 5A-5J, one such frame packing arrangement isshown that possesses some (or all) of the aforementioned advantages. Inthe context of RSP, the frame packing arrangement described subsequentlyherein may be referred to as RSP-3×2 or as RSP-2×3 (e.g., possessing a3:2 or 2:3 aspect ratio, respectively) as will become more readilyapparent from the following discussion. FIG. 5A illustrates a firstfacet 512 (or row) that may be obtained by, for example, directlycropping, for example, the middle portion from an ERP image 510. Forexample, the dashed line in FIG. 5A may be indicative of a croppedportion 512 of an ERP image 510 that covers a spherical image having,for example, 270° of angle coverage along the equator and 90° of anglecoverage along the meridian (e.g., a 3:1 aspect ratio for this croppedimage). As a brief aside, because ERP images can be thought of as havinga cylindrical shape, the cropped portion (indicated by the dashed lines)may be thought of as a partial cylinder. As illustrated in FIG. 5A, thecropped image 512 may be thought of as possessing the front, right andleft side views of a 360° spherical image.

Referring now to FIG. 5B, an alternative image of the same panoramicimage depicted in FIG. 5A is shown and described in detail. As shown inFIG. 5B, the ERP image has been re-projected such that the pixels at thepoles (e.g., top and bottom) have been rotated down to the equator.Additionally, the back side of the image in FIG. 5A has now been broughtto the front (i.e., at the center) in the image of FIG. 5B. Similar tothat shown with regards to FIG. 5A, the image 520 illustrated in FIG. 5Bmay be cropped (as indicated by the dashed white line) such that thecropped image 522 in FIG. 5B covers a spherical image having e.g., 270°of angle coverage along the equator and 90° of angle coverage along themeridian (e.g., a 3:1 aspect ratio for this cropped image) for thisre-projected image.

As a brief aside, and referring to FIG. 5C, this operation may bedescribed in terms of three-dimensional geometry 550. The depictedX-axis points toward the front view of the captured spherical image, thedepicted Y-axis points toward the top view of the captured sphericalimage, and the depicted Z-axis points toward the right view of thecaptured spherical image. Accordingly, when comparing the image depictedin FIG. 5B versus the image depicted in FIG. 5A, it may be seen that theimage of FIG. 5B possesses a 180° rotation along the Y-axis (e.g., tobring the back-side of the image to the front-side of the image) and a90° rotation along the X-axis (e.g., to bring the polar data(top/bottom) to the equator). While the aforementioned example should beconsidered exemplary, it would be readily appreciated by one of ordinaryskill given the contents of the present disclosure that other suitablerotations are also possible.

Referring now to FIG. 5D, the two cropped images 512, 522 from FIGS. 5Aand 5B are combined into a single image 500 (i.e., the top row may bethe cropped image of FIG. 5A while the bottom row may be the croppedimage of FIG. 5B). It would be readily appreciated by one of ordinaryskill given the contents of the present disclosure that other suitableframe packing arrangements may be substituted (e.g., the top row may bethe cropped image of FIG. 5B while the bottom row may be the croppedimage of FIG. 5A). In one or more implementations, the two images 512,522 may be obtained from a direct spherical to RSP mathematicalcalculation as set forth in Appendix I that forms part of the presentdisclosure. In other words, in some implementations it may not berequired to convert the spherical captured imaging data to, for example,an ERP image prior to transforming the image to the RSP imaging dataillustrated in FIG. 5D. Rather, the RSP image of FIG. 5D may begenerated directly from the captured imaging data using a mapping (e.g.,a pixel-to-pixel mapping) of the captured image format to the RSP imageformat.

FIG. 5E is a graphical representation 560 of an exemplary implementationof the depicted images of FIG. 5D oriented in their as-viewed state. Inother words, the arrangement depicted in FIG. 5E can be thought of aspossessing a similar geometry as the stitched portion of a baseball. Forexample, a full (or nearly full) 360° image may be represented using theframe packing arrangement depicted in, for example, FIG. 5D.Additionally, the number of discontinuities associated with the imagedepicted in FIG. 5D as compared with the number of discontinuitiesdepicted in, for example, FIG. 1C has been reduced by approximately 19%thereby achieving the goal of reducing the number of seams(discontinuities) associated with the frame packing arrangement of FIG.5D and hence may give better coding performance than the alternativearrangement of FIGS. 1A, 1B, and 1C.

The RSP-3×2 arrangement may allow for a reduction of pixels by 25% ascompared with a 360° video representation in an ERP format. For example,a 4K (4,096 pixels by 2,048 pixels) ERP coded image may only take 3,072pixels×2,048 pixels in the RSP-3×2 arrangement. The lower number ofpixels needed to represent the video data may result in improvedcompression efficiency, improved battery life, lower playback latencyand/or lower memory footprint needs.

Additionally, the RSP-3×2 arrangement of FIG. 5D may get closer to a16:9 aspect ratio as compared with the 6:1, 1:6, 8:3, and 3:8 framepacking arrangements previously depicted. Accordingly, line buffers maybe utilized more efficiently. Moreover, the RSP-3×2 arrangement of FIG.5D can signal 25% more resolution (or a reduction in transmissionbandwidth) as compared with a 360° ERP image, while still maintainingcompatibility with, for example, existing codecs. For example, theRSP-3×2 arrangement of FIG. 5D may enable a resolution of 3,744 pixelsby 2,496 pixels which would be compatible with Level 5.1 and Level 5.2constraints for H.264/AVC. In addition, the aforementioned 3,744 pixelsby 2,496 pixels may be readily decodable by already deployed hardwaredecoders. For example, the hardware decoders in many extant smartphonescan decode 3,744 pixels×2,496 pixel resolutions because the aspect ratiogets closer to 16:9 with the width being less than 3,840 pixels (e.g.,most smartphones are designed to decode and encode 4K 16:9 video (3,840pixels×2,160 pixels)). FIG. 5F and FIG. 5G depict alternativearrangements 570, 580 where the frame packing arrangement of FIG. 5D isrotated by 90° and 270°, respectively.

As a brief aside, and referring again to FIG. 5D, it may be seen thatredundant imaging data is contained within the top facet 512 and thebottom facet 522. For example, the person on the ski lifts right elbowis shown in both the bottom right portion of the bottom facet 522 aswell as the upper left portion of top facet 512. Accordingly, in someimplementations it may be desirable to “black out” portions of theRSP-3×2 (or RSP-2×3) image, thereby resulting in fewer bits for theframe of imaging data thereby resulting in improved compressionefficiency during the encoding process as well as fasterencoding/decoding of the image. FIG. 5H illustrates one such exemplaryimplementation for the blacking out of redundant imaging information fora frame 590 of imaging data. As shown, various portions 524 of the imagehave been blacked out. Specifically, in the frame 590 of imaging dataillustrated in FIG. 5H, the corners of both top facet 512 and bottomfacet 522 have been blacked out which is indicative of redundantinformation.

Referring now to FIG. 5I, another exemplary implementation for theblacking out of redundant imaging information for a frame 592 of imagingdata is illustrated. Similar to that shown with regards to FIG. 5H,various portion 524 of the image have been blacked out. However, unlikethe embodiment illustrated in FIG. 5H, the position of these blacked outportions 524 of the image do not reside at the corners of the top 512and bottom 522 facets. Rather, these blacked out portions 524 residebetween approximately 180° as indicated in the top facet 512 (i.e.,between −90° and +90°) and on the top and bottom of each of the top 512and bottom 522 facets. In some implementations, the approach shown inFIG. 5H may be preferred, as this particular configuration simplifiesthe drawing of the black arc while moving all black regions to thesides. In some implementations, the approach shown in FIG. 5I may bepreferred since it creates a natural black separation between the topand bottom facets, thereby mitigating artifacts that may arise as aresult of inter-facet filtering or motion compensation duringcompression, pre-processing or post-processing.

Referring now to FIG. 5J, yet another exemplary implementation for theblacking out of redundant imaging information for a frame 594 of imagingdata is illustrated. Unlike the embodiments illustrated in FIGS. 5H and5I, the top facet 512 does not include any blacked out imaginginformation; however, the bottom facet 522 includes a larger blacked outportion 524 than either of the frames 590, 592 of informationillustrated in FIGS. 5H and 5I. Specifically, the bottom facet 522 hasblacked out portions that correspond generally to the blacked outportions in FIG. 5H and the blacked out portions that correspondgenerally to the blacked out portion in FIG. 5I. Such an implementationmay have advantages where, for example, it may be advantageous topreserve all of the imaging information contained within the top facet512 (e.g., the top facet 512 may contain imaging information that ismore frequently associated with common viewport positions when viewingthis panoramic content), while the bottom facet 522 may have a fewernumber of bits (e.g., information) thereby resulting in fasterencoding/decoding of the information contained within the bottom facet522 of imaging data and/or improved compression efficiencies as comparedwith, for example, the top facet 512 imaging information. Additionally,the top facet 512 may include, for example, the blacked out portionsthat correspond generally to the blacked out portions in FIG. 5H and theblacked out portions that correspond generally to the blacked outportion in FIG. 5I, while the bottom facet 522 does not include anyblacked out portions in some implementations. These, and otherimplementations, would be readily apparent to one of ordinary skillgiven the contents of the present disclosure.

Additionally, any adjacent pixel resolution that may be available inthese blacked out portions may be used for the aforementioned metadatainformation that may, for example, be utilized for the stitching of thecaptured panoramic for display on a computing device. Additionally, oralternatively, in some implementations, these regions may includeinformation that enables a decoder to determine the particular encodingformat configuration chosen such as, for example, determining whichframe packing configuration of FIG. 5D, FIG. 5F, FIG. 5G, etc. has beenutilized for the encoding process. In some implementations, the darkregion (blacked out regions) may include additional information such asthat disclosed in co-owned and co-pending U.S. patent application Ser.No. 15/406,175 filed Jan. 13, 2017 entitled “Apparatus and Methods forthe Storage of Overlapping Regions of Imaging Data for the Generation ofOptimized Stitched Images”, incorporated supra. In one or moreimplementations, a decoder may utilize information contained within thedark regions (blacked out regions) for the decoding/rendering of theframe, while legacy decoders, for example, may simply discard any of theadditional data contained within these dark regions.

While the RSP embodiments described with reference to FIGS. 5A-5J havebeen primarily described in the context of using ERP imaging data, itwill be appreciated that the term RSP more broadly encompasses anyprojection that uses some portions of the image from an originalnon-rotated image (using any projection), while using other portions ofthe original non-rotated image after applying a spherical rotation andre-projection onto the original or any other projection as discussedsupra. Accordingly, in some implementations it may be desirable to use aso-called equal area adjustment to an equirectangular projection (ERP),or a native equal area projection (EAP) when encoding an RSP image(e.g., an RSP-3×2 arrangement or RSP-2×3 arrangement).

As a brief aside, although ERP has the advantage of being bothstraightforward and intuitive when displaying ERP imaging data (in theRSP format) on many common display devices, in some implementationsusing ERP imaging data (in the RSP format) may prove sub-optimal forvideo transmission in terms of bandwidth, image compression, bit rateconsiderations and/or imaging quality. Specifically, ERP maps sphericalcoordinates to a latitude and longitude grid with equal spacings. Thus,the top and bottom facets (or portions) of an ERP have adisproportionate number of pixels for encoding/compressing the image asa function of the latitude deviation from the equator. In other words,the geometric distortions introduced by ERP allocate more bits torepresent image data that is farther from the equator. These areas alsocontain the most distorted pixel information for the ERP image data;consequently, video compression and encoding quality metrics at suchlocations take more bandwidth to represent worse image quality.

Referring now to FIG. 5K, one such exemplary illustration of an ERP isshown and described in detail. In the illustrated frame 595 of ERPimaging data, the various circles/ellipses 596, 597, 598 arerepresentative of the number of pixels required as a function oflatitude in an ERP image (and are depicted for purposes of illustratingthe effects of ERP distortion and aren't necessarily drawn to scale).For example, areas resident on the equator (e.g., a latitude of 0°) arerepresented by small circles 596. At greater latitudinal deviations fromthe equator, the number of pixels required for the encoding of the samesized circle increases. For example, at a latitude of +15°, the area ofellipse 597 is larger than the small circle 596, thereby requiring anincrease in the number of pixels required for encoding at this latitude.Similarly, at a latitude of −15°, the area of ellipse 597 is also largerthan the small circle 596, thereby requiring an increase in the numberof pixels required for encoding. At a latitude of +30°, the area ofellipse 598 is even larger than both the area of ellipse 597 and thesmall circle 596, thereby requiring an even larger number of pixels forencoding. At a latitude of greater than 45° (not shown), the area of theellipse may be even greater (more pronounced) than that depicted by, forexample, ellipse 598, thereby requiring additional pixels for encoding.

In contrast, equal area projection (EAP) maps spherical coordinates to alatitude with equal spacings, but a longitude with spacings thatdecrease at the poles corresponding to the curvature of the sphere. Inother words, EAP image information at the higher latitudes are stretchedlatitudinally and/or shrunk longitudinally by a commensurate amount,such that the bits used to represent an area of the image aresubstantially equivalent between the equal area and sphericalprojections. EAP may have better video compression and encoding qualitymetrics when compared to ERP in some implementations, however EAPtypically appears more distorted to human perception in, for example,wider fields of view. For example, in implementations in which a widerfield of view is encoded (e.g., 360°×180° (i.e., +/−90°)), ERP maycommonly have better video compression and encoding quality metrics thanEAP. However, in implementations such as the implementation depicted inFIG. 5K (i.e., +/−45°), the introduced distortion may be significantlyless pronounced than a wider field of view format and hence an EAP mayhave better video compression and encoding quality metrics than ERP.Accordingly, depending upon the implementation chosen, it may or may notbe desirable to implement an equal area adjustment to, for example, anERP portion of an RSP imaging format.

Referring now to FIG. 5L, the image of FIG. 5K is represented in an EAPmapping. In the illustrated frame of EAP imaging data 595, the variouscircles/ellipses 596, 597, 598 are representative of the number ofpixels required as a function of latitude in an EAP image. At greaterlatitudinal deviations from the equator, the number of pixels requiredfor the encoding of the same sized circle stays constant because theellipse is stretched latitudinally and shrunk by a commensurate amountlongitudinally.

While EAP preserves image bandwidth (i.e., the information energyassociated with each image pixel) across the entire imageproportionately, artisans of ordinary skill in the related arts willreadily appreciate that still other projections may further adjust thenumber of bits at different latitudes to achieve desirable effects. Forexample, a band of latitudes at the center of the projection may bepresented according to ERP, whereas bands at larger latitudesprogressively shift to EAP. In one such implementation, it may bedesirable to adjust the heights (and/or widths) of ellipses 597, suchthat the height of these ellipses 597 is the same as the height of smallcircle 596 (thereby preserving the human perception advantages of ERP).However, it may be desirable to adjust the heights (and/or widths) ofthe ellipses 598 such that the respective areas of these adjustedellipses 598 is the same as that of ellipses 596 (thereby improvingcompression). Accordingly, applying an equal area adjustment to an ERPimage in its whole or a portion thereof, can be used to e.g.: (i)improve compression efficiencies associated with the encoding process,(ii) reduce the bandwidth/bit-rate required for imaging datatransmission, and/or (iii) improve the imaging quality associated withthe imaging data. These and other implementations would be readilyapparent to one of ordinary skill given the contents of the presentdisclosure.

Referring now to FIG. 6A, one exemplary methodology 600 for the creationof a target panoramic projection (e.g., an RSP-3×2 arrangement orRSP-2×3 arrangement) is shown and described in detail. At step 602,panoramic content is obtained in a source projection. In one or moreimplementations, the source projection may include the imaging datacaptured from an image capture device in its source sphericalprojection.

At step 604, the obtained panoramic content is split into a firstportion and a second portion. In some implementations, the first portionmay consist of the front, right and left portions of the originalpanoramic content, while the second portion may consist of the back, topand bottom portions of the original content. In other implementations,the first portion may consist of the front, back and right portions ofthe original panoramic content, while the second portion may consist ofthe top, left and bottom portions of the original panoramic content. Inyet other implementations, other portions of the original panoramiccontent may be arbitrarily selected so long as there is continuitywithin each of the selected portions. In other words, these selectedportions may be representative of any continuous portion of the obtainedpanoramic content. These and other implementations would be readilyapparent to one of ordinary skill given the contents of the presentdisclosure.

At step 606, a first target projection is selected for the first portionand a portion of the image is cropped from the first target projectionin order to create a first cropped portion. In some implementations, thefirst target projection may be an ERP projection and the first croppedportion may be representative of, for example, 270° of coverage alongthe equator and 90° of coverage along the meridian. In otherimplementations, the representative coverage may include other angularcoverages (e.g., 260°, 280° or other selected degree of coverage alongan arbitrarily selected equator and 80°, 100° or other selected degreeof coverage along an axis that is orthogonal to the arbitrarily selectedequator). In some implementations, step 606 may provide a directspherical to RSP mathematical calculation (e.g., cropping may beunnecessary) as set forth in, for example, Appendix I. These and otherimplementations would be readily apparent to one of ordinary skill giventhe contents of the present disclosure.

At step 608, a second target projection is selected for the secondportion and a portion of the image is cropped from the second targetprojection in order to create a second cropped portion. In someimplementations, the second target projection may be a portion of an RSPprojection (e.g., the back face is brought to the face and the poles arerotated to the equator) and the second cropped portion may berepresentative of, for example, 270° of coverage along the equator and90° of coverage along the meridian. In other implementations, therepresentative coverage may include other angular coverages (e.g., 260°,280° or other selected degree of coverage along an arbitrarily selectedequator and 80°, 100° or other selected degree of coverage along an axisthat is orthogonal to the arbitrarily selected equator). In someimplementations, step 608 may provide a direct spherical to RSPmathematical calculation (e.g., cropping may be unnecessary) as setforth in, for example, Appendix I. These and other implementations wouldbe readily apparent to one of ordinary skill given the contents of thepresent disclosure.

At step 610, the first cropped portion is combined with the secondcropped portion in order to create a target panoramic projection. Insome implementations, the target panoramic projection may include an RSPprojection as discussed previously herein with regards to, for example,FIG. 5D. The combined portions may be oriented geometrically asdiscussed with regards to FIG. 5D, may be oriented geometrically asdiscussed with regards to FIG. 5F, or may be oriented geometrically asdiscussed with regards to FIG. 5G. In some implementations, the targetpanoramic projection may include a non-RSP projection consisting of twoimage facets. For example, one facet may consist of an ERP projectionthat covers a first hemisphere of a spherical coordinate system (thatignores the poles), while the second facet may consist of an ERPprojection that covers the other hemisphere of a spherical coordinatesystem (that ignores the poles). These and other implementations wouldbe readily apparent to one of ordinary skill given the contents of thepresent disclosure.

Referring now to FIG. 6B, one exemplary methodology 650 for the decodingand display of a target panoramic projection (e.g., an RSP-3×2arrangement or RSP-2×3 arrangement) is shown and described in detail. Atstep 652, the target panoramic projection created at, for example, step610 may be received. In some implementations, the received targetpanoramic projection may include a first cropped portion and a secondcropped portion.

At step 654, a viewport position of interest associated with the targetpanoramic projection is received. In one or more implementations, thereceived viewport position may be oriented entirely within the firstcropped portion, may be oriented entirely within the second croppedportion, or may be oriented such that the imaging information for thefirst and second cropped portions are both needed for the receivedviewport position.

At step 656, the first cropped portion and/or the second cropped portionare decoded in accordance with the received viewport position, while theremainder of the frame of imaging data may, for example, simply bediscarded. Such an implementation may be desirable as the processingresources associated with the decoding process may be reduced ascompared with a decoding process in which the entirety of the receivedtarget panoramic projection. In some implementations, the entirety ofthe received target panoramic projection may be decoded using, forexample, a portion of the encoding/decoding apparatus illustrated inFIG. 7 and described subsequently herein, without taking intoconsideration the received viewport position.

At step 658, the decoded target panoramic projection (or decodedportions thereof) are transmitted to a display device and thetransmitted portion(s) are displayed on a display device at step 660. Inimplementations in which the first cropped portion and/or the secondcropped portion are decoded in accordance with the received viewportposition, only the decoded portion of the received target panoramicprojection may be transmitted at step 658, thereby reducing the amountof bandwidth required for the transmission. In implementations in whichthe entirety of the received target panoramic projection is decoded, theentirety of the decoded target panoramic projection may be transmittedat step 658 and displayed at step 660. In some implementations, thedisplay device may include a VR display, such as that described abovewith reference to, for example, FIG. 2. In other implementations, thedisplay device may include a display for a smart phone, a monitor of acomputing device or television, and/or other types of display devices.

Encoding/Decoding Apparatus for Two Facet Frame Packing Arrangements—

Referring now to FIG. 7, an exemplary encoding/decoding apparatus 700for two facet frame packing arrangements (e.g., an RSP-3×2 frame packingarrangement, an RSP-2×3 frame packing arrangement and the like) is shownand described in detail. The encoding/decoding apparatus includes animage capture device 702; a stitching module 704; two encoders 706, 708;a transmission line 710; two decoders 712, 714; a reconstruction module716; and a display device 718. While illustrated as a unitary system700, it is appreciated that individual portions of the encoding/decodingapparatus 700 may be implemented in separate hardware modules. Forexample, the image capture device 702 may constitute a unitarycomponent; the stitching module 704 and encoders 706, 708 may constitutea unitary component; the transmission line 710 may constitute a unitarycomponent; the decoders 712, 714 and reconstruction module 716 mayconstitute a unitary component; and the display device 718 mayconstitute a unitary component. These, and other implementations, wouldbe readily apparent to one of ordinary skill given the contents of thepresent disclosure.

The image capture device 702 may be configured to capture panoramiccontent. In some implementations, the image capture device 702 mayinclude two camera components (including a lens and imaging sensors)that are disposed in a Janus configuration, i.e., back to back such asthat described in U.S. patent application Ser. No. 29/548,661, entitled“MULTI-LENS CAMERA” filed on Dec. 15, 2015, the foregoing beingincorporated herein by reference in its entirety. In one or moreimplementations, the image capture device 702 may include a six-axiscamera device such as that described in U.S. patent application Ser. No.15/432,700 filed Feb. 14, 2017 and entitled “Apparatus and Methods forImage Encoding using Spatially Weighted Encoding Quality Parameters”,the contents of which are incorporated herein by reference in itsentirety. The image capture device 702 may be configured to capturestatic content and/or may be configured to capture video content.

The stitching module 704 is configured to receive the panoramic contentcaptured by the image capture device 702. In some implementations, thereceived panoramic content is composed of two or more images obtainedfrom respective ones of the imaging sensors of the image capture device702. These images may have overlapping fields of view with other one(s)of the captured images. Moreover, in some implementations, the stitchingmodule 704 may be embodied within the image capture device 702. Theoutput of stitching module 704 may include two image facets (such as,for example, the aforementioned RSP-3×2 frame packing arrangement orRSP-2×3 frame packing arrangement).

In the illustrated embodiment, each image facet may be fed into arespective encoder 706, 708. For example, where the output of thestitching module 704 is an RSP-3×2 frame of imaging data, the top imagefacet may be fed into encoder A 706, while the bottom image facet may befed into encoder B 708. Herein lies one salient advantage of theencoding/decoding apparatus 700 illustrated in FIG. 7. Namely, as thetop image facet and the bottom image facet may have identical resolutionwith each of these image facets having perfect continuity, the use oftwo separate encoders 706, 708 may allow for a more efficient encodingof imaging data by, inter alia, effectively doubling the throughput aswell as improving upon the compression efficiencies of the encoders 706,708. In some implementations, the output of the stitching module 704 mayinclude two (or more) image facets with disparate numbers of pixels suchas that described with reference to FIGS. 8A-8E described infra.Additionally, while FIG. 7 illustrates two encoders (i.e., encoder A 706and encoder B 708), in some implementations the number of encoders maybe increased (e.g., three encoders for the frame packing arrangement ofFIG. 4B may be utilized, for example, e.g., one encoder for the leftimage facet 406, one encoder for the right image facet 410, and oneencoder for the polar imaging data included in facets 408, 412 as butone example).

Transmission line 710 may provide for a transmission path for theencoded data coming from, for example, encoder A 706 and encoder B 708.Transmission line 710 may include a wired transmission line (e.g., anEthernet cable) and/or may include a wireless transmission line (e.g.,using a Wi-Fi transmission protocol and/or other wireless transmissionprotocols). In some implementations, the output of encoders 706, 708 maybe combined prior to transmission on transmission line 710. For example,top image facet 512 may be combined with bottom image facet 522 into asingle frame of imaging data 500 prior to transmission over transmissionline 710. In other implementations, the output of encoders 706, 708 maybe transmitted over transmission line 710 as separate bitstreams (e.g.,stored as separate tracks inside a container file). These and otherimplementations would be readily apparent to one of ordinary skill giventhe contents of the present disclosure.

In the illustrated embodiment, each image facet may be received by arespective decoder 712, 714. For example, where the imaging informationreceived over transmission line 710 is an RSP-3×2 frame of imaging data,the top image facet may be fed into decoder A 712, while the bottomimage facet may be fed into decoder B 714. Herein lies yet anothersalient advantage of the encoding/decoding apparatus 700 illustrated inFIG. 7. Namely, the effective doubling of the throughput for, forexample, decoders 712, 714. In some implementations, the transmission ofimaging information over transmission line 710 may include two (or more)image facets with disparate numbers of pixels such as that describedwith reference to FIGS. 8A-8E described infra. Additionally, while FIG.7 illustrates two decoders (i.e., decoder A 712 and decoder B 714), insome implementations the number of decoders may be increased (e.g.,three decoders for the frame packing arrangement of FIG. 4B may beutilized, for example, by using one decoder for the left image facet406, one decoder for the right image facet 410, and one decoder for thepolar imaging data included in facets 408, 412, as but one example).

Reconstruction module 716 may receive the output of the decoders 712,714. Additionally, in some implementations reconstruction module 716 mayalso receive an input that is indicative of a viewport of interest forthe obtained panoramic content. In implementations in whichreconstruction module 716 receives a viewport of interest, the imagingdata associated with the received viewport position may be output todisplay device 718 into a desired projection suitable for display ondisplay device 718. In some implementations, the entire panoramiccontent is reconstructed into a spherical format (or other desiredprojection format such as ERP, icosahedron, cube map and the like) andoutput to display device 718.

Non-Uniform Encoding Quality and/or Spatial Resolution Considerations

In some implementations, it may be desirable to alter the encodingquality for various ones (or portions) of the imaging facets, and/ordownsample various ones (or portions) of the imaging facets. Forexample, when reproducing panoramic content on a display device, it maybe desirable to maintain the imaging quality for areas of interest.These areas of interest may include a received viewport position, or maybe representative of a probability for a future area of interest (e.g.,when viewing static panoramic content, imaging along an equatorial planeand/or other areas of interest). Conversely, it may be desirable todecrease the imaging quality and/or downsample the imaging content inother areas of the captured panoramic content in certain areas. Forexample, if it is determined (or expected) that a certain area withinthe panoramic imaging content is of decreased importance, it may bedesirable to decrease the imaging quality and/or downsample theseregions prior to transmission over, for example, transmission line 710in FIG. 7. While the embodiments of FIGS. 8A-8E will be primarilydiscussed in the context of exemplary RSP-3×2 frame packingarrangements, it would be readily appreciated by one of ordinary skillgiven the contents of the present disclosure that the techniquesdescribed below may be equally applied to other frame packingarrangements such as, for example, the frame packing arrangementsdepicted in FIGS. 1A-1C and 3A-4F.

Referring to FIG. 8A, an exemplary RSP-3×2 frame packing arrangement 800is illustrated having top imaging facet 802 and bottom imaging facet804. For example, the top imaging facet 802 may be determined to be ofhigher importance and may be encoded at a higher imaging quality by, forexample, lowering the quantization parameter (QP) for the pixels in thetop imaging facet 802 during the encoding process. In someimplementations, the bottom imaging facet 804 may be determined to be oflower importance and may be encoded at a lower imaging quality by, forexample, increasing the QP for the pixels in the bottom imaging facet804 during the encoding process, or vice versa. Additionally, in someimplementations it may be desirable to alter the imaging quality by, forexample, adjusting QP within a portion of an imaging facet. For example,where the top imaging facet 802 is indicative of the front, right andleft portions of panoramic content, it may be desirable to increase theimaging quality (lower QP) for the portion of the imaging facetassociated with the front of the image, while decreasing the imagingquality (higher QP) for the portions of the imaging facet associatedwith right and left portions of the panoramic content, for example. Insome implementations, this may be accomplished using slices or tiles ofthe type known in the image processing arts.

Referring to FIG. 8B, an exemplary RSP-3×2 frame packing arrangement 820is illustrated having top imaging facet 802 and bottom imaging facet806. In the illustrated embodiment, the bottom imaging facet 806 isdownsampled vertically by a factor of two. In other words, bysub-sampling the bottom facet vertically (i.e., keeping the same width,but reducing the height by a factor of two for the bottom imaging facet806), the 3:2 aspect ratio of the frame packing arrangement has beenaltered into a 2:1 aspect ratio for the frame packing arrangement 820.Accordingly, as the aspect ratio is now 2:1 (which is closer in value toan aspect ratio of 16:9), the frame packing arrangement 820 of FIG. 8Bmay be more suitable from commonly (widely) deployed extant imagingcodecs.

Referring to FIG. 8C, exemplary RSP-3×2 frame packing arrangements 830,840 are illustrated. For example, in frame packing arrangement 830, thebottom imaging facet 808 has been downsampled vertically by a factor oftwo as well as downsampled horizontally by a factor of two. The bottomimaging facet 808 may be split and rearranged into bottom imaging facet810. Accordingly, frame packing arrangement 840 will now have a 12:5aspect ratio. As compared with the frame packing arrangement in FIG. 8B,the pixel count of FIG. 8C has been reduced by 16.6%. Moreover, thenumber of pixels has been reduced by 53% as compared with an ERPprojection panoramic image while maintaining the same resolution as theERP projection in, for example, 270° of coverage along the equator and90° of coverage along the meridian (i.e., for the imaging informationcontained within the top imaging facet 802).

In the exemplary context of RSP-3×2 frame packing arrangements, it maybe desirable to maintain the imaging quality along a given equatorialplane (e.g., front, left, back and right side portion of a panoramicimage), while de-emphasizing the portions of the images located at thepoles (e.g., top and bottom portions of a panoramic image). In someinstances, the vast majority of the interesting objects of interest maybe captured along the given equatorial plane. FIGS. 8D and 8E illustrateone exemplary methodology for accomplishing this emphasis along a givenequatorial plane. Specifically, FIG. 8D illustrates an RSP-3×2 framepacking arrangement 850 where the bottom facet 804 has been divided upbetween the two polar regions 812, 816 and the back equatorial region814. In frame packing arrangement 860, the polar regions 812, 816 havebeen downsampled vertically by a factor of two as well as downsampledhorizontally by a factor of two resulting in imaging facets 818, 822.Imaging facet 814 has been maintained in frame packing arrangement 860.In frame packing arrangement 870 as illustrated in FIG. 8E, thedownsampled polar region facets 818, 822 have been placed adjacent toone another, while the equatorial imaging facet 814 has been split upand rearranged into imaging facets 814 a, 814 b. Frame packingarrangement 870 has a 2:1 aspect ratio, while preserving the imagingresolution along all of a given equatorial plane. However, as the levelof discontinuity has been decreased as compared with the frame packingarrangements 820, 840 in FIGS. 8B and 8C, in some implementations thecompression efficiency for frame packing arrangement 870 of FIG. 8E maybe less than that of frame packing arrangements 820, 840.

Referring now to FIG. 9A, one exemplary methodology 900 for theadjustment of one or both of the two image facets quantization parameteris shown. At step 902, panoramic content is obtained in a rotated sphereprojection and at step 904, a determination is made as to whether toadjust the quantization parameter for one (or portions thereof) for oneof the two image facets. As a brief aside, a rotated sphere projectionis generally composed of two facets with each facet having goodcontinuity within the respective facet. In some implementations, it maybe desirable to adjust the quantization parameter such that the valuevaries spatially throughout the facet. For example, the quantization maybe varied higher or lower, depending upon a determined (or anticipated)area of interest for the panoramic content. Other considerations mayinclude adjusting the quantization parameter as a function of theobjects contained therein. For example, relatively simple objects (e.g.,the sky, objects in shade and/or other homogenous objects where visualdetail may be deemphasized) may have their QP increased, while morevisually complex objects or other objects of interest may have their QPdecreased (increase in quality). For example, the use of superpixelssuch as that described in co-owned and co-pending U.S. patentapplication Ser. No. 15/251,896 filed Aug. 30, 2016 and entitled“Apparatus and Methods for Video Image Post-Processing for CorrectingArtifacts”, the contents of which being incorporated herein by referencein its entirety, may be utilized in conjunction with the determinationfor whether or not to alter QP values within portions of an imagingfacet.

At step 906, a determination is made as to whether to adjust thequantization parameter for the other one (or portions thereof) for oneof the two image facets using, inter alia, one or more of the techniquesdescribed with reference to step 904. At step 908, one or both of thetwo image facets (or respective portions thereof) QP values may beadjusted. Accordingly, through the adjustment of the QP values within,for example, a rotated sphere projection, various compressionefficiencies may be enhanced (increased) or de-emphasized (decreased)dependent upon various considerations as would be readily apparent toone of ordinary skill given the contents of the present disclosure.

Referring now to FIG. 9B, an exemplary methodology 950 for decreasingthe pixel size of frames for a rotated sphere projection (such as, forexample, RSP-3×2 frame packing arrangements, RSP-2×3 frame packingarrangements and other RSP frame packing arrangements) is shown. At step952, panoramic content is obtained in a rotated sphere projection. Atstep 954, a determination is made whether to downsample one or moreportions of the rotated sphere projection. For example, some or all ofthe considerations considered with reference to FIGS. 8B-8E may be takeninto consideration when determining whether to downsample one or moreportions of the rotated sphere projection. If it is determined, thatdownsampling shouldn't be performed, the obtained panoramic content inthe rotated sphere projection is transmitted at step 958. If it isdetermined that downsampling should be performed, at step 956 it isdetermined whether the downsampled one or more portions should berearranged. In some implementations, when downsampling is performedhorizontally (in addition to, or alternatively from verticaldownsampling), it may be desirable to rearrange the downsampled one ormore portions in accordance with line buffer limitations, continuityconsiderations and/or frame packing height considerations and the like.Rearrangement may also be performed according to the content containedwithin a frame and may be varied as the content varies (i.e. on aper-frame basis).

Exemplary Apparatus—

FIG. 10 is a block diagram illustrating components of an examplecomputing system able to read instructions from a computer-readablemedium and execute them in one or more processors (or controllers). Thecomputing system in FIG. 10 may represent an implementation of, forexample, an image/video processing device for encoding and/or decodingof the various frame packing arrangements as discussed with respect toFIGS. 1A-4G, 5D-5J, and 8A-8E or performing.

The computing system 1000 can be used to execute instructions 1024(e.g., program code or software) for causing the computing system 1000to perform any one or more of the encoding/decoding methodologies (orprocesses) described herein. In alternative embodiments, the computingsystem 1000 operates as a standalone device or a connected (e.g.,networked) device that connects to other computer systems. The computingsystem 1000 may include, for example, a personal computer (PC), a tabletPC, a notebook computer, or other device capable of executinginstructions 1024 (sequential or otherwise) that specify actions to betaken. In another embodiment, the computing system 1000 may include aserver. In a networked deployment, the computing system 1000 may operatein the capacity of a server or client in a server-client networkenvironment, or as a peer device in a peer-to-peer (or distributed)network environment. Further, while only a single computer system 1000is illustrated, a plurality of computing systems 1000 may operate tojointly execute instructions 1024 to perform any one or more of theencoding/decoding methodologies discussed herein.

The example computing system 1000 includes one or more processing units(generally processor apparatus 1002). The processor apparatus 1002 mayinclude, for example, a central processing unit (CPU), a graphicsprocessing unit (GPU), a digital signal processor (DSP), a controller, astate machine, one or more application specific integrated circuits(ASICs), one or more radio-frequency integrated circuits (RFICs), or anycombination of the foregoing. The computing system 1000 also includes amain memory 1004. The computing system 1000 may include a storage unit1016. The processor 1002, memory 1004 and the storage unit 1016 maycommunicate via a bus 1008.

In addition, the computing system 1000 may include a static memory 1006,a display driver 1010 (e.g., to drive a plasma display panel (PDP), aliquid crystal display (LCD), a projector, or other types of displays).The computing system 1000 may also include input/output devices, e.g.,an alphanumeric input device 1012 (e.g., touch screen-based keypad or anexternal input device such as a keyboard), a dimensional (e.g., 2-D or3-D) control device 1014 (e.g., a touch screen or external input devicesuch as a mouse, a trackball, a joystick, a motion sensor, or otherpointing instrument), a signal capture/generation device 1018 (e.g., aspeaker, camera, and/or microphone), and a network interface device1020, which also are configured to communicate via the bus 1008.

Embodiments of the computing system 1000 corresponding to a clientdevice may include a different configuration than an embodiment of thecomputing system 1000 corresponding to a server. For example, anembodiment corresponding to a server may include a larger storage unit1016, more memory 1004, and a faster processor 1002 but may lack thedisplay driver 1010, input device 1012, and dimensional control device1014. An embodiment corresponding to an action camera may include asmaller storage unit 1016, less memory 1004, and a power efficient (andslower) processor 1002 and may include multiple camera capture devices1018.

The storage unit 1016 includes a computer-readable medium 1022 on whichis stored instructions 1024 (e.g., software) embodying any one or moreof the methodologies or functions described herein. The instructions1024 may also reside, completely or at least partially, within the mainmemory 1004 or within the processor 1002 (e.g., within a processor'scache memory) during execution thereof by the computing system 1000, themain memory 1004 and the processor 1002 also constitutingcomputer-readable media. The instructions 1024 may be transmitted orreceived over a network via the network interface device 1020.

While computer-readable medium 1022 is shown in an example embodiment tobe a single medium, the term “computer-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storethe instructions 1024. The term “computer-readable medium” shall also betaken to include any medium that is capable of storing instructions 1024for execution by the computing system 1000 and that cause the computingsystem 1000 to perform, for example, one or more of the methodologiesdisclosed herein.

Where certain elements of these implementations can be partially orfully implemented using known components, only those portions of suchknown components that are necessary for an understanding of the presentdisclosure are described, and detailed descriptions of other portions ofsuch known components are omitted so as not to obscure the disclosure.

In the present specification, an implementation showing a singularcomponent should not be considered limiting; rather, the disclosure isintended to encompass other implementations including a plurality of thesame component, and vice-versa, unless explicitly stated otherwiseherein.

Further, the present disclosure encompasses present and future knownequivalents to the components referred to herein by way of illustration.

As used herein, the term “computing device”, includes, but is notlimited to, personal computers (PCs) and minicomputers, whether desktop,laptop, or otherwise, mainframe computers, workstations, servers,personal digital assistants (PDAs), handheld computers, embeddedcomputers, programmable logic device, personal communicators, tabletcomputers, portable navigation aids, J2ME equipped devices, cellulartelephones, smart phones, personal integrated communication orentertainment devices, or literally any other device capable ofexecuting a set of instructions.

As used herein, the term “computer program” or “software” is meant toinclude any sequence or human or machine cognizable steps which performa function. Such program may be rendered in virtually any programminglanguage or environment including, for example, C/C++, C #, Fortran,COBOL, MATLAB™, PASCAL, Python, assembly language, markup languages(e.g., HTML, SGML, XML, VoXML), and the like, as well as object-orientedenvironments such as the Common Object Request Broker Architecture(CORBA), Java™ (including J2ME, Java Beans), Binary Runtime Environment(e.g., BREW), and the like.

As used herein, the terms “integrated circuit”, is meant to refer to anelectronic circuit manufactured by the patterned diffusion of traceelements into the surface of a thin substrate of semiconductor material.By way of non-limiting example, integrated circuits may include fieldprogrammable gate arrays (e.g., FPGAs), a programmable logic device(PLD), reconfigurable computer fabrics (RCFs), systems on a chip (SoC),application-specific integrated circuits (ASICs), and/or other types ofintegrated circuits.

As used herein, the term “memory” includes any type of integratedcircuit or other storage device adapted for storing digital dataincluding, without limitation, ROM. PROM, EEPROM, DRAM, Mobile DRAM,SDRAM, DDR/2 SDRAM, EDO/FPMS, RLDRAM, SRAM, “flash” memory (e.g.,NAND/NOR), memristor memory, and PSRAM.

As used herein, the term “processing unit” is meant generally to includedigital processing devices. By way of non-limiting example, digitalprocessing devices may include one or more of digital signal processors(DSPs), reduced instruction set computers (RISC), general-purpose (CISC)processors, microprocessors, gate arrays (e.g., field programmable gatearrays (FPGAs)), PLDs, reconfigurable computer fabrics (RCFs), arrayprocessors, secure microprocessors, application-specific integratedcircuits (ASICs), and/or other digital processing devices. Such digitalprocessors may be contained on a single unitary IC die, or distributedacross multiple components.

As used herein, the term “camera” may be used to refer withoutlimitation to any imaging device or sensor configured to capture,record, and/or convey still and/or video imagery, which may be sensitiveto visible parts of the electromagnetic spectrum and/or invisible partsof the electromagnetic spectrum (e.g., infrared, ultraviolet), and/orother energy (e.g., pressure waves).

It will be recognized that while certain aspects of the technology aredescribed in terms of a specific sequence of steps of a method, thesedescriptions are only illustrative of the broader methods of thedisclosure, and may be modified as required by the particularapplication. Certain steps may be rendered unnecessary or optional undercertain circumstances. Additionally, certain steps or functionality maybe added to the disclosed implementations, or the order of performanceof two or more steps permuted. All such variations are considered to beencompassed within the disclosure disclosed and claimed herein.

While the above detailed description has shown, described, and pointedout novel features of the disclosure as applied to variousimplementations, it will be understood that various omissions,substitutions, and changes in the form and details of the device orprocess illustrated may be made by those skilled in the art withoutdeparting from the disclosure. The foregoing description is of the bestmode presently contemplated of carrying out the principles of thedisclosure. This description is in no way meant to be limiting, butrather should be taken as illustrative of the general principles of thetechnology. The scope of the disclosure should be determined withreference to the claims.

What is claimed:
 1. A method for encoding a panoramic image, the methodincluding: obtaining a first equirectangular projection that includes afront panoramic portion, a right-side panoramic portion, and a left-sidepanoramic portion; cropping the first equirectangular projection tocreate a first cropped portion; obtaining a second equirectangularprojection that includes a back panoramic portion, a top panoramicportion, and a bottom panoramic portion; cropping the secondequirectangular projection to create a second cropped portion; combiningthe first cropped portion with the second cropped portion in order tocreate a panoramic projection such that the panoramic projectioncomprises a single line of discontinuity between the first croppedportion and the second cropped portion, the single line of discontinuityforming a boundary resultant from a juxtaposition of the first croppedportion and the second cropped portion in a frame packing arrangement;and blacking out portions of the panoramic projection, the portions ofthe panoramic projection being blacked out comprising redundant imagingdata; nulling at least a portion of the panoramic projection, the atleast one portion of the panoramic projection that has been nulledcomprising redundant imaging data; wherein the single line ofdiscontinuity comprises a seam formed by one side of the firstprojection and a corresponding side of the second projection.
 2. Themethod of claim 1, further comprising: receiving a viewport position forthe panoramic projection, the viewport position being indicative of aportion of the panoramic projection; determining that an entirety of theviewport position is located in either the first cropped portion or thesecond cropped portion; decoding either the first cropped portion or thesecond cropped portion based on the determining; and transmitting eitherthe decoded first cropped portion or the decoded second cropped portion.3. The method of claim 2, further comprising: based at least on thedetermining, causing display of either the transmitted decoded firstcropped portion or the transmitted decoded second cropped portion. 4.The method of claim 1, wherein the blacking out the portions of thepanoramic projection comprises blacking out corners of the first croppedportion and the second cropped portion.
 5. The method of claim 1,wherein the blacking out the portions of the panoramic projectioncomprises blacking out portions internal to corners of the first croppedportion and the second cropped portion.
 6. The method of claim 1,wherein the blacking out the portions of the panoramic projectioncomprises only blacking out the second cropped portion of the panoramicprojection, while not blacking out the first cropped portion of thepanoramic projection.
 7. The method of claim 1, further comprisinginserting metadata information into the portions that have been blackedout of the panoramic projection.
 8. A non-transitory computer-readablestorage apparatus, the computer-readable storage apparatus comprising astorage medium having at least one computer program stored thereon, theat least one computer program comprising computer-readable instructions,the computer-readable instructions being configured to, when executed bya processor apparatus, cause a computerized apparatus to: obtain a firstprojection that includes a front panoramic portion, a right-sidepanoramic portion, and a left-side panoramic portion; obtain a secondprojection that includes a back panoramic portion, a top panoramicportion, and a bottom panoramic portion; combine the first projectionwith the second projection in order to create a panoramic projectionsuch that the panoramic projection comprises a single line ofdiscontinuity between the first projection and the second projection;and nulling at least a portion of the panoramic projection, the at leastone portion of the panoramic projection that has been nulled comprisingredundant imaging data; wherein the single line of discontinuitycomprises a seam formed by one side of the first projection and acorresponding side of the second projection.
 9. The non-transitorycomputer-readable storage apparatus of claim 8, wherein thecomputer-readable instructions are further configured to, when executedby the processor apparatus, cause the computerized apparatus to: receivea viewport position for the panoramic projection, the viewport positionbeing indicative of a portion of the panoramic projection; determinethat the entirety of the viewport position is located in either thefirst projection or the second projection; decode either the firstprojection or the second cropped portion based on the determination; andtransmit either the decoded first projection or the decoded secondprojection.
 10. The non-transitory computer-readable storage apparatusof claim 9, wherein the computer-readable instructions are furtherconfigured to, when executed by the processor apparatus, cause thecomputerized apparatus to: based at least on the determination, causedisplay of either the transmitted decoded first projection or thetransmitted decoded second projection.
 11. The non-transitorycomputer-readable storage apparatus of claim 8, the panoramic projectionusing a frame packing arrangement configured to lower a number ofdiscontinuities as compared to another frame packing arrangementcharacterized by a different aspect ratio.
 12. The non-transitorycomputer-readable storage apparatus of claim 8, wherein thecomputer-readable instructions are further configured to, when executedby the processor apparatus, cause the computerized apparatus to: insertmetadata information into the nulled at least portion of the panoramicprojection.
 13. An encoder apparatus, the encoder apparatus comprising:an image capture device, the image capture device configured to capturepanoramic content; a stitching module configured to generate a firstprojection that includes a front panoramic portion, a right-sidepanoramic portion, and a left-side panoramic portion from the capturedpanoramic content, the stitching module further configured to generate asecond projection that includes a back panoramic portion, a toppanoramic portion, and a bottom panoramic portion from the capturedpanoramic content; a first encoder configured to encode the firstprojection; a second encoder configured to encode the second projection;wherein at least a portion of the first projection comprises imagingdata redundant to the second projection, wherein nulling of the at leastportion of the first projection by the encoder apparatus is configuredto reduce a number of pixels associated with a combined image based on(i) the first projection comprising the nulled at least portion and (ii)the second projection, as compared with another combined image based on(i) the first projection without the nulled at least portion and (ii)the second projection; wherein the combined image comprises a singleline of discontinuity between the first projection and the secondprojection, the single line of discontinuity comprising only one seamformed by the first and second projections there between.
 14. Theencoder apparatus of claim 13, wherein the first projection isrepresentative of a first continuous portion of the captured panoramiccontent.
 15. The encoder apparatus of claim 14, wherein the secondprojection is representative of a second continuous portion of thecaptured panoramic content.
 16. The encoder apparatus of claim 13,wherein the encoder apparatus is further configured to insert metadatainformation in the nulled at least portion.
 17. The encoder apparatus ofclaim 16, wherein the inserted metadata information is utilized for astitch of the captured panoramic content for display on a computingdevice.
 18. The encoder apparatus of claim 16, wherein the insertedmetadata information is utilized for selection of a particular encodingformat configuration of a plurality of encoding format configurations,the combined image being configured as an image encoded based on theselected particular encoding format configuration.